System and method for video image registration in a heads up display

ABSTRACT

A system and method for aligning video images with an underlying visual field are provided. A video camera is coupled with a heads up display, and a computer positions images from the video camera on the heads up display based on the relative orientations of the camera and the display. As the video camera moves with respect to the display, the images are repositioned within the heads up display. The video image, which may, for example, come from a weapon sight, is aligned within the heads up display so that an observer can easily view the camera image without having to shift focus from the larger scene.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.11/000,934, titled “System and Method for Video Image Registration in aHeads Up Display” and filed Dec. 2, 2004, now U.S. Pat. No. 7,787,012,which application is incorporated by reference herein in its entirety.

FIELD OF THE INVENTION

The invention relates generally to the display of video images. Moreparticularly, the invention provides a method and system for registeringa video image with an underlying visual field, such as in a heads updisplay.

BACKGROUND OF THE INVENTION

Modern warfare has seen its share of technological improvements whichhave led to weapons that can be targeted with ever increasing levels ofspeed and accuracy, enabling weapon operators to react more quickly whena situation suddenly changes. While tanks, jets, missiles, combatplanning systems, and other technological implements have kept pace withmodern electronics, some familiar tools of modern warfare have remainedvirtually unchanged for centuries. Perhaps foremost among theseessential components is the infantryman: the soldier carrying light armsdeployed on foot.

Infantrymen have benefited to some extent from modern technology withthe advent of laser sights, night vision goggles, and so forth. Thesehave allowed the foot soldier to navigate at night, and accuratelydispatch their targets. These technologies ultimately help to keep thesoldier safe under the cover of darkness, help give the element ofsurprise when needed, and also help ensure that the first shot fired isthe one that hits the mark.

In spite of these advances, one problem which persists is the cumbersomeprocess of acquiring and striking a target. At night, modern nightvision goggles passively amplify miniscule amounts of ambient light,such as starlight, and enable a soldier to see obscured targets in thedark. Once a target is found in this fashion, however, a soldier mustflip the goggles out of the way and reacquire the target with the sighton his weapon. This takes time away from the soldier, during which hemight be seen by the target itself, or the target might move. Inaddition, reacquisition with the weapon's narrower field of vision maybe virtually impossible with a distant or moving target.

Alternatively, a soldier, upon viewing a target with night visiongoggles, may engage a laser illuminator on his weapon. The illuminatorprojects a beam of laser light following the line of sight of the weaponand striking where the bullet will strike. The soldier can keep hisgoggles on and see the illuminated point. He can move the point of theilluminator until it points to his target, and then fire as needed.While somewhat faster than lifting the goggles and reacquiring thetarget through the weapon sight, the illuminator may have the unintendedeffect of giving away the soldier's position. The laser illuminator maybe just as obvious to an enemy as it is to the soldier. In the time ittakes to maneuver his weapon into position, he may already be spottedand in the weapon sight of his enemy.

In the hopes of solving this and other problems inherent with currentinfantry technology, U.S. military planners have envisioned atechnological revolution for the foot soldiers of tomorrow, dubbedFuture Force Warrior. The project envisions, among other improvements,the porting of an infantryman's weapon sight into a heads up display(HUD) built into his night vision goggles. Such goggles exist now, as inSensor Technology Systems' Model 2733 Low Profile Night Vision Goggle.They have the ability to port a video feed into a beam combiner,overlaying a video image from a video source mounted in the weapon sightonto the center of the visual field of the goggles.

An example of such a combined image appears as prior art FIG. 1. Here,the video feed 102 from a weapon's sight is superimposed directly intothe center of the night vision goggle's visual field 101. This isaccomplished using a beam combiner, which optically overlays one imageover another. Both images depict the same subjects, a group of soldiersaccompanying an armored personnel carrier (APC). However, the video feed102 remains stationary in the center of the visual field 101, obscuringcontent in the center of the visual field, in this case the APC and asoldier. The two images are distinctly offset, with the two soldiers tothe right of the APC being repeated in both images. This offset, withtwo distinct images of the same target appearing in different places inthe field of view, could confuse the soldier, causing a delay inengagement or a miss. If the soldier moves his weapon and turns his headsimultaneously, the set of images moving in different directions may beeven more confusing and disorienting to the soldier, potentiallydecreasing the soldier's ability to react and the accuracy of any shotfired.

Thus, it would be an advancement in the art if a video image from avideo source could be integrated into the visual field of a heads updisplay without confusing or disorienting the observer, and withoutneedlessly obscuring relevant visual content.

SUMMARY OF THE INVENTION

A first embodiment of the invention provides a method for aligning videoimages with an underlying visual field by determining a sourceorientation of a video source, determining a display orientation of atransparent display overlaying the visual field, and displaying videoimages in the transparent display, wherein a position for the images isbased on the source orientation and the display orientation.

A second embodiment of the invention provides a system for displaying aportion of a video feed overlaying a visual field comprising a videocamera, a heads up display (HUD), and a computer. Orientation sensorsare affixed to the video camera and the HUD. The computer is adapted toreceive sensor data from both orientation sensors, to receive the videofeed from the video camera, and to display video images in the HUD.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a prior art example of a stationary video feedoverlaying a visual field.

FIG. 2 illustrates an example of a visual field produced by night visiongoggles.

FIG. 3 illustrates an example image from a video camera capturing thesame scene as the visual field of FIG. 2.

FIG. 4 illustrates the image produced by an illustrative embodiment ofthe invention.

FIG. 5 illustrates the system employed by an illustrative embodiment ofthe invention.

FIG. 6 is a block diagram which depicts the functional components of anillustrative embodiment of the invention.

FIG. 7 is a block diagram which depicts the functional components of acomputer employed by an illustrative embodiment of the invention.

FIG. 8 illustrates a method for registering a video image with anunderlying visual field.

FIG. 9A depicts an illustrative embodiment of the invention with a videosource and heads up display visually aligned.

FIG. 9B illustrates a combined image created by the illustrativeembodiment of FIG. 9A.

FIG. 10A depicts an illustrative embodiment of the invention with avideo source and heads up display visually offset horizontally.

FIG. 10B illustrates a combined image created by the illustrativeembodiment of FIG. 10A.

FIG. 11A depicts an illustrative embodiment of the invention with avideo source and heads up display visually offset vertically.

FIG. 11B illustrates a combined image created by the illustrativeembodiment of FIG. 11A.

FIG. 12A depicts an illustrative embodiment of the invention with avideo source and heads up display visually offset in rotation.

FIG. 12B illustrates a combined image created by the illustrativeembodiment of FIG. 12A.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 2 illustrates an example of a visual field 200 as seen through thenight vision goggles a foot soldier might wear. The image enhancesambient light, typically starlight, to enable the soldier to see intothe night. The visual field depicts a handful of potential targets forthe foot soldier. FIG. 3 illustrates an example image 300 from a videocamera capturing the same scene as the visual feed of FIG. 2. The videocamera producing the image 300 may be attached to a weapon carried bythe foot soldier. The image 300 may be the product of a specializedcamera or weapon sight, such as a thermal imager which makes infraredwavelengths visible, a starlight scope which amplifies ambient lightusing the same technology which enables night vision goggles, or anyother video source such as a standard television camera. The video feedincludes cross hairs 301 so that a soldier viewing the video feed willknow exactly where a shot fired will hit. In addition, the video cameramay magnify the image to aid target recognition and to increaseaccuracy.

The visual field 200 and the video image 300 differ in their field ofview (FOV). The visual field of the night vision goggles allows for agreater amount of visual information to be viewed by the observersimultaneously without the observer having to move his head. The FOV ofthe video image 300, as from a weapon sight, is normally much narrower,presenting less of the visual field to the observer. As a result, asoldier looking directly through a weapon sight, or looking at the videofeed produced by such a sight, will have to move the weapon in order toview the entire scene. For this reason, soldiers may search for andacquire targets at night using the wider FOV of night vision goggles,and switch to the weapon sight view only once they have decided upon atarget. This takes time, thus slowing down the soldier.

The visual field 400 of FIG. 4 illustrates the image produced by anillustrative embodiment of the invention. The visual field 400, here theview through a soldier's night vision goggles or other (clear) goggles,is enhanced with the addition of a portion of the weapon sight videofeed 401 through the use of a heads up display (HUD). With thisembodiment of the invention, the video feed 401 may be modified so as todiscard unneeded visual information, cropping the image to perhaps onequarter of its original size. In addition, the video feed 401 has beenpositioned over the portion of the visual field 400 based on thedirection the video source is pointed. As the weapon moves, the videofeed 401 is dynamically positioned within the visual field 400.Ultimately, by superimposing the two images, a soldier with a heads updisplay and a weapon mounted video camera is able to simultaneouslysurvey a setting, acquire a target, and point his weapon at the targetwithout taking time to shift from goggles to weapon sight.

It should be noted upfront that this superimposition of video images isnot limited to weapon usage on a battlefield, although that is thedominant example used here. Other embodiments of the current inventioncould be used in a myriad of settings, including law enforcement,medicine, etc. For example, a surgeon could use such a device on hishand to provide a magnified view of an operating field embedded within aview of the entire patient's chest cavity. An astronomer could survey astar field visually while wearing some form of heads up display. Hertelescope could be fitted with a video camera, the feed from which isdynamically fed into and positioned within the HUD. As such, she canview a magnified video image from the telescope without having toreposition herself before the telescope's eyepiece. Alternatively, hervideo source could produce a modified output, perhaps displaying acolor-shifted spectrographic view of the light from a particular star.Similarly, a nephologist can survey a sky full of clouds andsimultaneously focus in on clouds of particular interest withoutshifting. An ichthyologist, fitted with an underwater embodiment of theinvention, could survey a school of fish and simultaneously focus in ona particular fish. In each of these alternative embodiments, a secondaryvideo source is used to dynamically supplement an observer's field ofview.

FIG. 5 illustrates a system employed by an illustrative embodiment ofthe invention. Here, an infantryman 500 is fitted with goggles 505, arifle 502 with video gun sight 503, and field computer 501. The goggles505 may produce a visual field similar to the one illustrated in FIG. 2.The video gun sight 503 produces a video feed, possibly including crosshairs as in FIG. 3, depicting the line of sight of the rifle 502. Thevideo weapon sight 503 may produce more than a standard unmagnifiedview, for example a magnified view, a thermal view, a night vision view,an image intensifier view, or some combination thereof.

In this embodiment, field computer 501 receives a video feed from videoweapon sight 503 via cable 512. The video feed may be delivered usingany standard video format, for example analog formats like NTSC or PAL,or digital formats like MPEG, or any non-standard format. The fieldcomputer 501 receives sensor data from orientation sensors 504 and 506,via cables 510 and 511. Once the video feed is processed, field computer501 delivers video for the heads up display within the goggles 505, viacable 513.

The sensor 504 affixed to rifle 502 sends data relaying the orientationof the weapon and attached video gun sight 503. This data may includeangular pitch, yaw, and roll information, sent in frequent intervals. Anexample of such a sensor is InterSense's IntertiaCube3®, which uses theearth's gravitational and magnetic fields (among other means) to senseand report angular orientation around three axes of rotation up to 180times per second. The sensor 506 affixed to goggles 505 relays similarorientation data, except that it reports on the line of sight of thegoggles instead of the rifle 502. It should be noted that sensors 504need not be directly affixed to the rifle 502, so long as it moves withthe rifle. For example, it could be attached to the gun sight 503.Likewise, sensor 506 need not be directly affixed to the goggles 505.The sensor 506 could also be attached to the helmet of the infantryman500.

In this embodiment, sensors 504, 506 use Universal Serial Bus (USB)cables 510, 511 to relay angular data, although any communication methodis feasible. These cables, along with video cables 512, 513 may eitherbe exposed or sewn into a soldier's clothing or his rifle sling toprevent entanglement of the wearer. Although wired sensors and videocables are used here, any form of wireless radio is feasible. Forexample, Ultra-wideband (UWB) transceivers may transmit video and sensordata from the weapon, and sensor data from the goggles. Likewise, UWBmay be used to transmit video from the field computer 501 to the goggles505. Although UWB radios, such as Time Domain's PulsON® radio, areparticularly desirable for their high bandwidth, low power consumptionand for being virtually undetectable, any wireless standard may be used,including both Bluetooth and IEEE 802.11.

In alternative embodiments, UWB radios may be used for more thantransmission of video and sensor data. Multiple radios may be placed onthe rifle 502 and on the goggles 505 (or on the helmet, to which thegoggles may be affixed), each of which can relay their precise position.In this fashion, the field computer 501 may be able to calculate thealignment of the rifle and goggles based on the relative location ofradios rather than use separate orientation sensors.

In other alternative embodiments, the heads up display need not beconnected to the viewer, as through a pair of night vision goggles. Forexample, the heads up display could appear before a windshield in avehicle. A weapon mounted on the vehicle includes a video gun sightproducing images processed and projected onto the heads up display. Inthis embodiment, an orientation sensor may be placed to sense theorientation of the vehicle rather than a pair of goggles worn by theobserver. This embodiment may be particularly useful for remotelycontrolled weapon systems, for example a robot carrying a weapon. Thecurrent state of the art uses two screens, one for navigation and onefor aiming the weapon. A robot operator uses one screen to drive therobot and acquire targets, then refers to an adjacent screen to aim andfire the weapon. Registering the weapon video image to the navigationscreen in a manner similar to an infantryman garners similar advantagesfor the robot operator. Additionally, because a robot's weapon istypically turret-mounted, sensors may be replaced with similar gear orother positional readouts based on turret position, laser rangefinderposition, or weapon elevation position.

FIG. 6 is a block diagram which depicts the functional components of anillustrative embodiment of the invention. Here, computer 601 receivessensor data and a video feed from video assembly 604, along with sensordata from heads up display assembly 607. Video assembly 604 is composedof video source 602 and sensor 603 affixed to detect the orientation ofthe video source. Video source 602 has a visual field 611 from which itreceives light and converts it to the video signal delivered to computer601. Heads up display assembly 607 is composed of beam combiner 605 andsensor 606 affixed to detect the orientation of the beam combiner. Beamcombiner 605 has a visual field 610, whose image is combined with theprocessed video signal delivered from computer 601. This combination ofvideo signal with visual field may be created through the use of atransparent display, such as a piece of glass set at an angle. The glassmay pass light from the visual field 610 to the observer whilesimultaneously reflecting light from a video display strategicallyplaced based on the angle of the glass. The transparent display need notbe perfectly transparent, but also might be translucent allowing onlysome light to pass through. The video output of computer 601 is placedin front of the visual field 610, creating what is sometimes referred toas a heads up display or HUD. Such displays allow an observer to receiveinformation or images while simultaneously viewing a visual field,preventing the observer from having to look away.

FIG. 7 is a block diagram depicting the functional components of acomputer employed by an illustrative embodiment of the invention. Thefunctional components of computer 601 illustrated here are merelyrepresentative of functions. Individual functions may be combined ordivided among multiple components within the device. Here, processor 701is connected to memory 702 via bus 710. Memory 702 may include volatilememory, such as random access memory (RAM), or non-volatile memory, suchas Flash or a hard disk drive. Also connected to processor 701 isInput/Output Interface 704, which may communicate with and pass datafrom connected peripherals, including orientation sensors, perhaps usingUSB or a wireless standard, such as UWB or Bluetooth. Video interface705 receives video signals and relays them for storage in memory 702 orprocessing in processor 701. Display interface 703 relays video signalsto an external display, such as the HUD. Optional network interface 706may be used to communicate with an external computer, possibly totransmit and receive position and situational data (to other teammembers, or via satellite back to headquarters). Bus 710 may becomprised of a single or multiple signal buses for communicating betweencomponents.

FIG. 8 demonstrates an illustrative embodiment of a method forregistering a video image with an underlying visual field. It should benoted that the steps pictured here may be reordered, combined, or splitto achieve a similar result. Step 801 initiates the method when theheads up display is initiated, either through a switch attached to avideo source or gun sight, a switch on a computer, or perhaps on theheads up display itself. Alternatively, the display may be initiatedwhen a weapon is removed from its safety setting. Once initiated, atstep 802, a video frame is received for processing. The frame may beprocessed digitally, and if it is received in analog form may first needto be converted to a digital format for processing.

Along with the receipt of a video frame, orientation data may bereceived from sensors attached to a heads up display and a video source,as in step 803. This data may be received in the form of pitch, yaw, androll angular values or in quarternions. Such values indicate the angleof vertical rise (pitch), the angle of horizontal rotation (yaw), andthe angle of rotation around the line of sight (roll), for both thedisplay and the video source. Having this data, in step 804, thedifference in pitch and yaw values between the display and the videosource can be calculated. The pitch delta is the difference in pitchvalues from the two sensors, and the yaw delta is the difference in yawvalues. By knowing the pitch delta and yaw delta, the location of theprocessed frame within a heads up display is determined, as in step 805.In determining the location, the pitch delta and yaw delta values aremapped from degrees to pixels. This calculation requires determining thenumber of pixels in a degree of vision, and then multiplying that numberby the pitch delta and yaw delta values to determine vertical andhorizontal offset from the center of the visual field in pixels.

In step 806, the roll delta value is determined in similar fashion, byfinding the difference between the roll values sensed at the videosource and display. Based on the roll delta, the processed frame can berotated for presentation within the heads up display, as in step 807.Various algorithms for rotating an image by a certain number of degreesare well known in the art.

Once the location and rotation of the processed frame within the displayare determined, the frame may be cropped, discarding unneeded pixels, asin step 808. The frame may be resized in order to map the videoinformation onto the pixels that will ultimately be used in a heads updisplay. This step may be necessary if the video images produced by avideo source are larger than needed for display. For example, if a videoimage initially has a field of view of 8 degrees horizontal and 6degrees vertical, it may be cropped down to 4 degrees horizontal and 3degrees vertical, retaining the same center point. In this fashion, onlya quarter of the image is retained, but it constitutes the most relevantpart of the image. Alternatively, the video frame may need to bemagnified or compressed in order to adjust for differences inmagnification between the visual field and the native video frame. Inaddition, the frame may be enhanced by adding a border around the frameso as to further distinguish it from the visual field for an observer.

The processed video frame, at this point, may be displayed in a heads updisplay, as in step 809. The colored pixels of the processed frame aresurrounded by dark or black pixels, which equate to transparent in aheads up display. The displayed frame appears before a visual field fromthe perspective of an observer of the visual field. The calculatedposition and rotational orientation of the processed frame place it onthe display approximately in front of the same subject matter depictedin the visual field. In addition to the frame itself, additionalinformation may be added to the display, for example, battery life data.The final display image, including the processed and repositioned videoframe and any other information, is sent to the display, possibly usinga standard video format such as 12-bit Video or NTSC.

At this point in the process, at decision 810, if another frame of videois set to be received (i.e., the display is still on), then the processrepeats for each new frame, returning to step 802. In this fashion, eachframe of video is individually processed, modifying the frame,positioning and rotating it based on the difference in orientationsbetween the video source and the display, and then displaying it. Assuch, if the video source moves from left to right, then its orientationdata will change, and subsequent displayed frames will move left toright across the visual field, aligning or registering each frame withthe portion of the visual field it overlays. Once there are no longerany frames to be processed (i.e., the display has been turned off), theprocess comes to an end.

FIG. 9A depicts an illustrative embodiment of the invention in the formof a video source 912 affixed to rifle 911 and heads up display goggles901. The direction that rifle 911 and video source 912 are pointing issensed by orientation sensor 915. The line of sight for the video sourceis indicated by the Z-axis on axes 916. The direction that heads updisplay goggles 901 are pointing is sensed by orientation sensor 905.The line of sight for the goggles 901 is indicated by the Z-axis on axes906. Here, axes 906 and 916 are in alignment. The goggles are looking inexactly the same direction as the weapon is pointed. As such, theorientation sensors 905 and 915 will output the same values for pitch,yaw and roll.

The effect of this upon the heads up display is depicted in FIG. 9B,which illustrates a combined image created by the illustrativeembodiment of FIG. 9A. When a frame 921 from the video source 912 isprocessed using an embodiment of the invention, it is placed in thecenter of the visual field 920, as shown. Here, the visual field isdepicted with increments of degrees which represent the field of view,in this case through the heads up display goggles 901. The examplevisual field 920 for the goggles 901 has a field of view that is 32degrees horizontally (yaw) and 20 degrees vertically (pitch). If thepitch delta and yaw delta values are zero (i.e., the goggles and videosource are aligned), then the frame 921 is displayed in the center ofthe visual field 920, as shown here. The roll delta value is also zerohere, because the rifle 911 and goggles 901 are both rotationallyaligned with each other around their respective Z-axes. As such, theframe 921 is displayed without rotation.

FIG. 10A depicts the same components as FIG. 9A. Here, however, therifle 911 has been rotated horizontally by 8 degrees to the left. FIG.10B depicts the subsequent change in the visual field 920. The pitch androll values detected by orientation sensor 915 remain unchanged.However, the yaw value detected will change by 8 degrees to the left.When this is compared to the values detected by the orientation sensor905 affixed to goggles 901, which haven't changed, the yaw delta valuewill be −8 degrees. When processed, this yaw delta value will change theplacement of the frame 1021 in the heads up display, shifting it to theleft as shown. If the yaw value exceeds the visual field width, theframe may still appear within the visual field, but with a distinctiveborder, indicating to the user that the rifle is pointing outside thevisual field. The frame remains visible, however, giving the user a viewof where the weapon is pointing. The frame position will be in thedirection of the actual weapon pointing direction, allowing the user torapidly and instinctively adjust his visual field or his weapon to bringthe two back together.

FIG. 11A again depicts the same components as FIG. 9A. Here, the rifle911 has been rotated vertically (rather than horizontally) up 6 degrees.FIG. 11B depicts the subsequent change in the visual field 920. The yawand roll values detected by orientation sensor 915 remain unchanged, butthe pitch value detected will change, up 6 degrees. When compared to theunchanged values from the orientation sensor 905 affixed to the goggles901, the pitch value will be +6 degrees. When frame 1121 is processed,this pitch delta value will change the placement of the frame in theheads up display, shifting it up as shown.

FIG. 12A is a final depiction of the same components as FIG. 9A. Here,the rifle 911 has been rolled around its line of sight by 10 degrees,counterclockwise. FIG. 12B depicts the subsequent change in the visualfield 920. The pitch and yaw values detected by the rifle's orientationsensor 915 remain unchanged, but the roll value detected is different,counterclockwise 10 degrees. When compared to the unchanged values fromthe orientation sensor 905 affixed to the goggles 901, the roll deltavalue will be 10 degrees. When frame 1221 is processed, this roll deltavalue will change the rotation of the frame in the heads up display,rotating it counterclockwise. It should be noted that although FIGS.10A-12B depict only motion around a single axis at a time, the techniquemay be applied to motion in all directions.

One possible consideration for the above described methods and systemsis the problem of parallax. That is, because a video source and a headsup display are separated by some distance (e.g. 0.5 meters), if bothdevices are perfectly aligned, they will in fact be looking at slightlydifferent points. As a result, in processing a video frame, the locationwhere the frame is placed may be slightly off, and a displayed frame ofvideo will not be aligned as perfectly as possible. However, thisproblem diminishes as the distance to a target increases. The furtheraway the target, the smaller the change in degrees, and hence thesmaller the error produced. For example, a target at 10 meters with 0.5meters between gun sight and goggles produces an error of about 2.9degrees in the placement of the video frame. At 100 meters, with thesame 0.5 meters between gun sight and goggles, the error is only 0.29degrees in the placement of the video frame.

The problem of parallax is to some extent a non-issue. The systemproposed would likely be used for targets at distances greater than 10meters more often than not. Moreover, when targeting a weapon using thesystem, the video frame displayed in the heads up display willultimately be the source of assurance that a soldier's weapon is pointedat a proper target. Even if a video gun sight image is slightlymisaligned with the visual field surrounding it, the soldier willprimarily care that his weapon is pointed at the correct target.Further, the absolute amount of misalignment will be only 0.5 meters atworst using the above example. The video gun sight image will stillappear over or next to the intended target. Note that weapon accuracy isgoverned by zeroing the weapon and the video gun sight image, soparallax has no effect on bullet accuracy.

To the extent that parallax is an issue, it can be handled in severalways. One solution is to integrate a range finder, such as a laser rangefinder, into the system to automatically detect the distance of targetsand, given a known weapon-goggle distance, adjust the image placementaccordingly. Another solution is to provide a range adjustment controlwhich a soldier can use to manually adjust the range to a target, andaccordingly adjust image placement.

Parallax may also be an issue when calibrating the orientation sensors.Calibration may be required when initially configuring the invention,and may also be required if a severe jolt to the system causesmisalignment of the sensors. One solution may be to provide a buttonwhich is pushed to signal calibration. While holding the button, areticle may appear in the center of the visual field while the videoimage may appear motionless off to the side. Once the visual field andvideo image are both centered on the same target, releasing thecalibration button signals alignment to the system. The portion of theinvention which calculates the position of the video image may thenadjust its calculations accordingly. Centering on a distant target (forexample, greater than 300 meters) during the calibration routine may berequired as parallax will induce an angular error in closer targets, asdescribed above.

Alternative embodiments may provide additional solutions to the issue ofparallax. In one such embodiment, the image produced in the visual fieldof a display is captured by a second video camera. This second videofeed or goggle image, along with the video images from the video source,are both fed into a computer for initial digital processing. As such,well known rigid or non-rigid image registration techniques may be usedto register the images by, for example, finding common visual elementsbetween them. This process is accelerated by having, based on theorientation data, a starting point from which to search the goggleimage. Once the precise location of the video image is registered withinthe goggle image, the video image can be more accurately aligned. Thevideo image may then be displayed in the heads up display alone, or theheads up display may be filled with the resultant combination of videoimage and goggle image.

While the invention has been described with respect to specific examplesincluding presently preferred modes of carrying out the invention, thoseskilled in the art will appreciate that there are numerous variationsand permutations of the above described systems and techniques. Thus,the spirit and scope of the invention should be construed broadly as setforth in the appended claims.

We claim:
 1. A system comprising: a video camera adapted to provide, ina video feed, data for a series of video images representing portions ofa visual field; a first orientation sensor adapted to detect anorientation of the video camera; a heads up display (HUD) adapted forviewing of the visual field by a user of the system wherein the HUDcomprises a transparent display, and wherein the HUD and the videocamera are independently movable about multiple axes; a secondorientation sensor adapted to detect an orientation of the HUD; and acomputer adapted to receive sensor data from the first and secondorientation sensors, to receive the video feed from the video camera,and to display the video images, on the transparant display and based onthe received sensor data, in positions that overlay portions of thevisual field represented by the displayed video images whereinboundaries of the displayed video images are in registration withboundaries of portions of the visual field represented by the displayedvideo images, and wherein the computer is adapted to determine a sourceorientation of the video camera, and determine a display orientation ofthe transparent display.
 2. The system of claim 1, wherein the HUD ishoused in a pair of goggles.
 3. The system of claim 1, wherein the videocamera is attached to a weapon and the video feed is of a line of sightof the weapon.
 4. The system of claim 1, wherein the video camera is aweapon sight attached to a weapon, the first orientation sensor isaffixed to the weapon and, the computer is adapted to resize and cropthe video images.
 5. The system of claim 1, wherein the HUD is housed ina pair of night-vision goggles.
 6. The system of claim 5, wherein thevideo camera is attached to a weapon.
 7. The system of claim 6, whereinthe video camera is a thermal gun sight.
 8. The system of claim 1,wherein the video camera is a thermal gun sight.
 9. The system of claim1, wherein the computer is adapted to reposition the displayed videoimages within the transparent display when the video camera ortransparent display moves.
 10. The system of claim 9, wherein the videocamera is attached to a weapon.
 11. The system of claim 10, wherein thevideo feed is of a line of sight of the weapon.
 12. The system of claim1, wherein the computer is a field computer adapted to be worn andcarried by a human user.